Data Onboarding
Onboard new data sources to Splunk with GDI agent — generates deployment-ready config packages with Magic 8 compliance
Data Onboarding
The GDI Onboarding Agent analyzes your sample logs, matches CIM data models, and generates a complete deployment-ready Splunk app package — including serverclass.conf for deployment server distribution.
Navigation
- Parent: Use Cases
- Related: GDI Onboarding Agent | Integrations
The Scenario
You have a new data source to onboard to Splunk. You need inputs, parsing, field extractions, CIM alignment, and a way to deploy configs across forwarders, indexers, and search heads via deployment server. Doing this manually means writing configs by hand, validating Magic 8 compliance, and building Splunk app packages. The GDI agent automates the entire process.
What the Agent Generates
Configuration Files
| File | Purpose |
|---|---|
| inputs.conf | Monitor stanzas for forwarders — file paths, sourcetype assignment, index routing |
| props.conf | Index-time and search-time parsing — line breaking, timestamp extraction, Magic 8 compliance |
| transforms.conf | Field extractions, lookups, sed replacements — both index-time and search-time |
| tags.conf | CIM tagging — maps events to CIM data models (e.g., Authentication, Network Traffic) |
| serverclass.conf | Deployment server classes — host whitelists/blacklists, restart behavior, app-to-host mapping |
Multi-App Layout
The agent creates 4 Splunk apps per sourcetype, following best practices for deployment:
| App | Deploys To | Contains |
|---|---|---|
TA-{sourcetype}_inputs | Forwarders | inputs.conf |
TA-{sourcetype}_indexer | Indexers | props.conf, transforms.conf (index-time) |
TA-{sourcetype}_search | Search Heads | props.conf, transforms.conf (search-time), tags.conf |
TA-{sourcetype}_deployment | Deployment Server | serverclass.conf |
This structure separates concerns by Splunk tier and is ready for distribution via deployment server or SH Deployer.
Magic 8 Compliance
Every generated config is validated against the Magic 8 best practices:
- Clear sourcetype — unique, descriptive name
- Line breaking — handles multi-line events
- Timestamp extraction — correct format and timezone
- Key field extraction — consistent field names and types
- CIM alignment — mapped to relevant data models
- Validation — syntax and parsing verification
- Documentation — sourcetype ownership and description
- Monitoring — ingestion tracking post-deployment
Workflow
- Provide sample data — paste or upload a sample of the new data source.
- Describe the source — application name, format, expected volume, retention needs.
- Agent analyzes — inspects log format, matches CIM data models, identifies fields.
- Review generated configs — the agent produces the full multi-app package with all config files.
- Validate — the agent runs syntax validation and Magic 8 compliance checks.
- Push to GitHub (optional) — create a branch and PR with the generated configs for code review.
- Deploy — distribute via deployment server using the generated serverclass.conf.
- Monitor — use the Data Quality Check workflow after go-live.
Required Integrations
| Integration | Purpose |
|---|---|
| Splunk MCP | Inspect existing configs, validate against live environment |
| Regex for Splunk | Generate and test field extraction regex |
| GitHub (optional) | Push configs to a repo, create branch and PR for review |
| Deslicer Observer (optional) | Deploy configs via Observer API with approval workflows |
