When we started putting ETL Assistant together it became immediately clear that we needed a UI. Something to help us get the job done, but something that didn’t get in the way. We threw together a quick ASP.NET MVC 3 web app that mostly gets the job done.

Below you’ll find some UI samples from the web management console. This should help express a bit more of where we’ve gone and how the whole application functions. This is just to facilitate the configuration “stuff” we toss over to the SSIS package at runtime.

Couple of quick points here:

This was my first MVC app. I was still thinking in webforms and it shows. A lot.
Yes, we went with the ~~Metro~~ Win8 style UI
I took these screenshots on a Macbook Pro with retina display and I rushed through posting them. Please ignore the visual artifacts from my over-compressing things, etc.
This has evolved over time. Hindsight is 20/20 and if we were to start over from scratch this would probably look very different.

Jump to specific sections

Home & Global UI Elements

Home page
“Quick start,” high level stats, and recent items. We provide easy access to setting up new datasources and groups as well as adding tables to groups. This is a simple, wizard-like 3 step process. We also show some basic stats on the number of objects in the system. Yes – we really do use this for over 30,000+ tables across 30+ separate source systems.

Menu Example
Categorized mini-mega-menu with recent items and common functions. Just a simple way for us to provide easier access to key functions while saving on screen real-estate. The items on the right provide quick access to recently used data sources, groups, or tables (specific to each menu) as well as faster access to “create” functionality.

Search Bar
Inline global search with categorized results. The search system isn’t particularly smart, but it gets the job done and saves you some pain – otherwise you’d needed to “page” through lists of items.

Data Source Display & Editing

Creating – Basic settings
These are the basic connection settings, source/target schema settings, destination/staging filegroup settings, etc. required for ETL. The connection tester is here to help as well.

Selecting a Data Source Type
Data source type selection when creating / editing a data source. The data source selection defaults global behaviors so you don’t have to specify them for each and every data source (EX: “How do I query the source system catalog?”). We use OLEDB, so as long as you have the right driver and can configure the required settings it should pretty much work. Should…

Creating – Advanced settings
More fun settings. You can override the global data source defaults here as well as set some other options to throttle performance or enable super-high-risk features like table definition synchronization (MSSQL ONLY!). That’s right… For the ultra bleeding edge, highly volatile systems we let you auto-synchronize your target table definition against source table structure using SMO.

Testing a connection string
Determine your connectivity problems early on. We bubble up the underlying connection exception message if there’s an issue.

Testing a connection string
Green = good to go!

Detail – Basic settings
Review all that stuff you set up earlier.

Related Groups
Quick access to any “related” groups – groups which have tables that are bound to this data source. Just a simple helper to prevent forcing you to “hunt” for data. This, as with most tabs, is ajax-ified, so it only queries the system when this tab receives focus.

Scheduled Tables
Quick access to any tables (specific to this data source) which are known to be scheduled. This is arrived at by looking at the related groups and then querying the MSSQL job setup.

Retrieve & Refresh Table Catalog

Refresh table catalog
The UI allows for inline querying of the source system to detect any new/updated/removed tables

Refresh table catalog – Result
Once the system polls the source database you get a simple set of counts of new/updated/removed tables. Clicking on these panels / boxes shows you a quick list of objects.

Table Display & Editing

Basic Settings
Basic settings in display mode. Just like how the data source can override settings from the data source type defaults, the table can override settings set in the data source. Need to target a different schema or use a different staging database? No problem – just change things. This is particularly useful if your table requires a different full or incremental load query.

Basic / Advanced Settings
Basic and advanced settings in display mode. We provide even more bells & whistles here. You can set your table to stage even on full load (helpful if you don’t want to wipe out your data during loads), specify behavior to override Nick Smith’s dynamic merge (aka “Honey Badger”), enable table structure synch via SMO (dangerous x1000), or have MSSQL message you if “critical” columns disappear or change in the data source. “Oh, they changed the data type on employee_id? Let me know!”

Generate SQL
Generate table DDL. Another helper to simplify your life. This is embedded in a few places.

Groups Editing & Table Management

Group Detail
Poor, neglected UI that displays simple group information (name, etc.) as well as the link to any scheduled instances of this group.

Scheduled Tables
Display the list of tables scheduled for this group. You can search for tables via partial name match (scoped to a data source) and then add them as “full” or “incremental” loads. This may also be adjusted within this UI. Scheduled tables may be removed or disabled / enabled from within this UI as well.

DDL – 1/3
We provide a “bulk” DDL generator – to generate DDL for all objects within selected group. The DDL process uses a custom class to reflect on source / destination objects and then uses “Smart Column” configuration to override specific types as required.

DDL – 2/3
Part way through generation of DDL for all objects within selected group

DDL – 3/3
Done generating DDL for all objects within selected group

DDL – Final SQL
Result of “generate SQL” (DDL). This is just a subset of the output.

Table DDL Builder – “Smart Columns”
Override target data types by source system, data type, and column name (using regular expressions). Our philosophy is the data architect knows best. We let you specify replacement data types in several different ways.

SQL Server Jobs, Job History

Job Steps
List job steps for a system job. For the lazy (me), this helps give me a quick way to see what’s going on with ETL Assistant-related jobs without jumping to Management Studio.

Job Schedule
Provide information on the job schedule for a specified job. Pulling apart those bitmasks is super fun for the whole family.

Job History
Display summary job history for instances of this job. How are we looking?

Job History Detail
Display job history for a specific job. Expanded items use ajax (jquery + json) to pull in details as required. We only take what we need to survive.

Query Variables (QVars)

Example
Example of QVars in use for an incremental loading query. These are extremely handy for writing templates queries where you need to substitute values at ETL runtime. “Get me the latest timestamp for this table” and then replace that variable in the sql statement when we query the data source. There’s a lot more to this than we’re going to go into in a screenshot.

Drag / Drop
QVars support drag-and-drop from the QVar panel into specific destination panels. Hooray for jquery awesomeness!

Example – Data Source QVar
QVar panel displayed when editing a datasource or table. QVars shown are system-wide QVars, but then intelligently display “scoped” QVars appropriate / available to a specific datasource or table