Every data SOURCE we touch gets a folder under _datashare/ named for the source agency (e.g. TEA_AEIS, THECB, TDI). Inside every source folder you'll find:
01_raw/ — raw files exactly as the source provided them. Read-only.02_cleaned/ — cleaned analytic .dta files.03_output/ — tables, figures, exports for sharing._code/ — Stata do-files that read raw → write cleaned → write output._documentation/ — this website, READMEs, snapshotted source docs._archive/ — prior vintages worth keeping.Every datashare project ships with seven numbered do-files in _code/:
| File | Role | Writes to |
|---|---|---|
000_controlfile.do | Paths + globals + Stata environment | (globals only) |
100_data_download.do | Fetches raw files from the source | 01_raw/ |
200_data_management.do | Imports raw, cleans, reshapes, merges | 02_cleaned/ |
300_labels.do | Variable + value labels, notes, characteristics | 02_cleaned/ (in-place re-save) |
400_data_profiler.do | Codebook, summary stats, missingness, frequencies | (webdoc2 page only) |
500_aggregation.do | Collapse / contract to county-year, statewide, panel | 02_cleaned/ |
600_analysis.do | Headline figures + tables for this source | 03_output/ |
Each file is independently re-runnable — every do-file calls dsload at the top, which loads 000_controlfile.do if its globals aren't already in scope.
000_controlfile.do in turn loads 050_preferences.do (color palette, graph dimensions, number formats, default file types), so every numbered file inherits the same look and feel without restating preferences.
Each numbered do-file is also a webdoc2 document. Running it:
_documentation/website/.wdinit (Bootstrap 5 + Inter font + back-to-top button), draws a navbar with wdds_navbar, and an in-page TOC with wdtoc.button/buttonclose (collapsed by default) or wd/wdclose (always-visible log). Graphs use graphbutton/graphbuttonclose.Two extra do-files in _documentation/ finish the site:
index.do — builds index.html, the landing page (auto-inventories raw/cleaned/output)help.do — builds this pageEach project carries a small metadata block, set at scaffold time and editable any time:
| Global | Meaning | This project |
|---|---|---|
Insurance_PriceTransparency_BCBSTX | folder name (CamelCase, underscored) | Insurance_PriceTransparency_BCBSTX |
Transparency-in-Coverage machine-readable files (in-network negotiated rates + prescription drugs) published by Blue Cross and Blue Shield of Texas (HCSC). | one-line description | Transparency-in-Coverage machine-readable files (in-network negotiated rates + prescription drugs) published by Blue Cross and Blue Shield of Texas (HCSC). |
https://www.bcbstx.com/member/machine-readable-files | source agency URL | https://www.bcbstx.com/member/machine-readable-files |
health care, insurance, prices | topic tag(s) | health care, insurance, prices |
| yes / no / unsure | |
monthly refresh (source files restated ~monthly) | refresh cadence | monthly refresh (source files restated ~monthly) |
| free-text caveats | |
| up to 10 outcome vars for the auto-profiler | |
| up to 10 by-vars for the auto-profiler | |
| 1 = export descsave codebook in 300_labels |
After editing any do-file, rebuild the website with:
. do "/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/_documentation/_runall.do"
_runall.do calls webdoc2 file.do, replace cleanup for every numbered file, then builds index.html and help.html last. On Mac it opens index.html at the end.
To rebuild ONE page only:
. webdoc2 "/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/_code/400_data_profiler.do", replace cleanup open
TEA_AEIS, THECB, TDI — not topics or dates.2006_Discipline.dta, 2006_2010_Probation.dta._final, no initials, no timestamps in primary names — those belong in _archive/.Discipline_Enrollment_Tickets.dta._v1, _v2 only for true releases. Prior cuts go to _archive/.Full conventions: _datashare/_documentation/README_NamingConventions.txt
The Stata pipeline is canonical. To add work in another language, drop parallel files into _code/ using the same numeric step + a language-appropriate extension:
. * Equivalent steps in different languages . * 100_data_download.do Stata (canonical) . * 100_data_download.R R alternative . * 100_data_download.py Python alternative . * 200_data_management.ipynb Jupyter notebook . * 500_aggregation.qgz QGIS project . * 600_analysis_dashboard.html static dashboard built by 600
Outputs from non-Stata code still land in 02_cleaned/ (cross-language artifacts: .parquet, .csv, .feather) or 03_output/ (deliverables).
| Global | Resolves to |
|---|---|
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team | Google Drive root (Shared drives/Data and Research Team) |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_codeshare | Shared drives/Data and Research Team/_codeshare |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare | Shared drives/Data and Research Team/_datashare |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX | this project folder |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/01_raw / /Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/01_raw/_converted | raw files / machine-converted copies |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/02_cleaned | cleaned analytic tables |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/03_output | output deliverables |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/_code | do-files |
/Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/_documentation / /Users/ericbooth/Library/CloudStorage/GoogleDrive-eric.booth@texas2036.org/Shared drives/Data and Research Team/_datashare/Insurance_PriceTransparency/BCBSTX/_documentation/website | documentation / built website |
From any Stata session:
. projectbuilder MySource_Subsource, desc("Short description") url("https://source.example.com/")
This scaffolds _datashare/MySource_Subsource/ with the full folder tree, all seven do-files, and the documentation templates pre-filled. See help projectbuilder for options.