Datasets

Because datasets were built to be flexible and general-purpose, they can be easily configured to support a dizzying array of potential workflows. This page will walk you through a series of examples, so that you can get an idea for the possibilities.

Pre-loading data

Say you’re surveying clinics or households, and you already have some details from administrative data or a prior survey round. You can use datasets to pre-load that data into your survey form so that it might be put to good use.

Just create a new dataset, upload your data to populate it, and attach it to one or more of your survey forms. In your forms, you can then do handy things like ask users to choose a district, then a village, then a clinic or household, or you can verify the name of the head of household, automatically record the unique ID number for the household or clinic, etc.

Whenever your pre-loaded data changes, you can just upload revised or additional data – and any forms that use that data will automatically update.

Using pre-loaded data, you can pre-load survey team or enumerator assignments, the results of back-office randomizations, and more.

Connecting forms

Say that you’re using this pre-loading option to pre-load information gathered by a household listing team into your baseline survey forms. If you’re gathering the listing data with SurveyCTO, you can stream data directly from your listing forms into a dataset that is then attached to your baseline survey forms.

Just create a new dataset, configure your listing form to publish into it, and attach it to your baseline survey forms. Like with the manually-uploaded data discussed above, you can utilize that data to enrich your form in numerous ways – but unlike the case above, the form’s pre-loaded data updates automatically, as new data comes in.

Whenever new listing data is submitted, it streams directly into the dataset and automatically triggers an update of any forms that use that dataset.

If you only want to publish a subset of your data for pre-loading, you can do that too. So, for example, you could publish only a random tenth of your listed households for baseline surveying, or publish only a fifth of surveyed households for supervisor back-checks.

And your most sensitive data remains totally safe. If you want to publish some data for pre-loading but want to safely encrypt the rest with your own public-private key pair, you can just configure your form to be encrypted and specially flag those particular fields that are “publishable”; only those fields that you flag will be available for streaming into datasets, and the rest will remain “for your eyes only” (decrypted on export, optionally on a cold-room computer).

Monitoring data quality

The sooner you catch problems in your data, the easier and less costly they are to fix. With datasets, you can easily configure quality checks to automatically watch for outliers, suspicious distributions, too many missing values, and more. You can then receive nightly email reports that flag critical errors, suspicious surveys, or enumerators whose responses differ significantly from the rest of the team.

Just configure one or more forms to publish data you’d like monitored into a dataset, and then turn on quality checks for that dataset – or just configure quality checks directly for one or more forms. Then configure what warnings to issue under what circumstances, and to what email addresses you’d like nightly reports delivered. You can watch for out-of-bounds values or means, values that are too frequent or not frequent enough, means or distributions that differ by group when they should be uniform, and more. All with not a bit of programming.

The full set of warnings generated by your quality checks are themselves streamed into a dataset, which can then be downloaded, merged into Google or Excel spreadsheets, and more.

These automated quality checks make up an important part of SurveyCTO’s unprecedented mix of quality-control features. See our page on collecting quality data to learn more…

Managing data-collection operations

It can be a complex management challenge to oversee all but the simplest of data-collection activities. While SurveyCTO can’t manage your survey operation for you, it can at least meet you half way.

Say that you track your operations using spreadsheets in Microsoft Excel or Google Sheets. If you do, you can automatically publish a subset of your incoming data into those spreadsheets. SurveyCTO won’t disrupt any extra columns or rows that you have in those spreadsheets, so your team’s tracking updates and other edits won’t be lost when new data arrives.

Just configure one or more forms to publish a key subset of submitted data into a dataset, and then export that dataset to an Excel file or Google Sheet – or just export directly from one or more forms. Every time you export new data, it’ll merge with whatever’s already there in the spreadsheet, taking care not to lose anything.

If you want to fully encrypt your survey data with your own public-private key, you can still configure the dataset to publish on your local computer, post-decryption and -export. That option won’t work for Google Sheets, but it will for Excel.

Sharing data

You might want to publish some of your data to Google Sheets not just for internal tracking purposes, but for monitoring, sharing, publishing, visualizing, or any number of other purposes. Once your data is in Google Sheets, there’s pretty much no limit to what you can do with it.

Just configure one or more forms to publish selected fields to a dataset, then configure that dataset to publish to Google Sheets – or just configure Google publishing directly for one or more forms. As form data is submitted to the server, it will stream right out to Google Sheets.

Of course, you can still fully encrypt all of your most sensitive data and only publish a subset of fields to Google Sheets. You can also specify that only a subset of submissions be published; for example, you might want to only publish data for adults, or for respondents who specifically consented to their data being published more widely.

Monitoring and visualizing data

Finally, you might want to use Google Fusion Tables to monitor or visualize your data as it comes in, so SurveyCTO has that option too.

Just configure one or more forms to publish selected fields to a dataset, then configure that dataset to publish to Google Fusion Tables – or just configure Google publishing directly for one or more forms. As form data is submitted to the server, it will stream right out to Google, and any views or visualizations will update automatically.

Endless combinations

Multiple forms can publish into a single dataset. Datasets can be attached to multiple forms. Forms can attach multiple datasets. All the while, datasets can be configured to publish to Google Sheets and Fusion Tables. They can be exported to .csv, Excel, Google Earth, and more. Truly, the combinations are endless. All without any programming.