<div>Blumetra Solutions is a specialized software Product engineering firm founded in 2020, headquartered in the San Francisco Bay Area with a development center in Hyderabad, India. We partner with our Customers across retail, financial services, high tech, pharmaceuticals, and other industries to help them achieve their digital transformation objectives. We define and deliver business solutions combining the power of Data, Analytics, Business Applications, and Cloud Capabilities. We work with customer business and technical stakeholders to deploy and evangelize “adoption-ready” solutions and change management plans, with clearly understood and measured business value.</div>

Blumetra

Job Title: Databricks – Lead Data Engineer
Work Location: Hyderabad (Hybrid)
Experience: 7 – 10 years
&nbsp;
Role summary
We are looking for a hands-on Databricks Lead to design, build and take to production governed financial data products on an Azure Databricks lakehouse. You will own the Databricks solution end to end – file-based ingestion, medallion pipelines, hierarchy modelling, row-level security, zero-outage publishing and performance – and lead a small engineering team while working closely with source-system owners, finance stakeholders, and security and platform teams.
&nbsp;
What you will build
A typical engagement publishes multi-dimensional financial data (for example a management P&amp;L with entity, department, project, product, version, period and account dimensions) from an enterprise planning system as a single, row-secured consumption table in Databricks, refreshed several times a day. The quality bar:
<ul>
<li>Exact reconciliation to source control totals; no data loss.</li>
<li>Full fidelity of hierarchies, alternate hierarchies, aliases and attributes.</li>
<li>Zero consumer outage during refresh; a failed run never replaces the last good data.</li>
<li>Row-level security that matches the source system's access rules.</li>
<li>Sub-second response for agreed benchmark queries.</li>
<li>Additive schema changes absorbed without code changes; full rebuild from Git.</li>
</ul>
&nbsp;
Key responsibilities
Architecture and technical leadership
<ul>
<li>Own the Databricks solution design and drive open decisions with stakeholders: landing zone, publish mechanism, orchestration pattern, compute, retention and recovery targets.</li>
<li>Translate integration specifications into buildable designs, data contracts and a delivery plan; lead design reviews with integration architects and platform teams.</li>
<li>Lead, mentor and review the work of 2–4 data engineers; set coding, testing and documentation standards.</li>
<li>Raise risks early (data volume, security parity, BI connectivity) and propose options with trade-offs.</li>
</ul>
Platform setup and governance (Unity Catalog)
<ul>
<li>Set up catalogs, schemas, storage credentials, external locations and volumes for landing zones and medallion layers across dev, test and prod.</li>
<li>Define the grant model with account-level groups synced from Microsoft Entra ID; configure service principals for jobs and CI/CD.</li>
<li>Work with Azure network teams on private endpoints for ADLS and Network Connectivity Configurations for serverless compute.</li>
</ul>
Ingestion and orchestration
<ul>
<li>Build manifest-gated, file-arrival-triggered Databricks Workflows with task dependencies, task values, condition-task publish gates, queueing, retries, timeouts and notifications.</li>
<li>Implement Bronze ingestion from delimited files with checksum and row-count verification against a JSON manifest.</li>
<li>Guarantee idempotent reruns and safe handling of overlapping or late runs.</li>
</ul>
Data modelling and transformation (Silver / Gold)
<ul>
<li>Build PySpark logic to validate and flatten ragged parent-child hierarchies, detecting cycles, orphans, multi-parent members and non-unit weights.</li>
<li>Handle alternate hierarchies through configuration, as flattened tables or bridge tables.</li>
<li>Pivot long-form attributes and aliases into columns; build a contract-driven Gold generator that absorbs additive schema changes automatically.</li>
<li>Design wide, denormalized consumption tables, including leaf/consolidated grain flags to prevent double counting.</li>
</ul>
Data quality and reconciliation
<ul>
<li>Implement blocking quality checks (contract, referential integrity, hierarchy, typing, enrichment, security) and exact reconciliation to source control totals.</li>
<li>Build control tables (run log, check results, reconciliation, schema contract) and an operations dashboard.</li>
</ul>
Security
<ul>
<li>Implement source-system security parity: map source users to Entra identities, resolve group and member-level rights into a security lookup table, and enforce them with a Unity Catalog row-filter function.</li>
<li>Keep data and security consistent during publish; define break-glass access, ownership and audit monitoring.</li>
<li>Advise BI teams on per-user connectivity (SSO / DirectQuery) so row-level security holds end to end.</li>
</ul>
Publishing and performance
<ul>
<li>Implement zero-outage publishing with atomic Delta overwrites or blue/green swaps, plus rollback via time travel.</li>
<li>Tune for sub-second queries: liquid clustering, data-skipping statistics, predictive optimization, Photon, SQL Warehouse sizing and row-filter cost; run benchmark tests with real access profiles.</li>
</ul>
DevOps, testing and operations
<ul>
<li>Package everything as Databricks Asset Bundles with CI/CD (GitHub Actions or Azure DevOps) across dev, test and prod.</li>
<li>Own the test strategy: unit tests, contract and fixture tests, integration, SIT reconciliation, security UAT and performance tests.</li>
<li>Write the runbook (rerun, rollback, full rebuild, schema change) and hand over to production support.</li>
</ul>
Collaboration
<ul>
<li>Agree file contracts, manifests and control totals with source-system administrators.</li>
<li>Work with finance users to define benchmark queries and validate numbers during UAT and parallel runs.</li>
<li>Report progress, risks and decisions to stakeholders and delivery leadership.</li>
</ul>
&nbsp;
&nbsp;
Required skills

Area
Databricks core
Unity Catalog
Fine-grained security
Delta Lake
Performance
PySpark and SQL
Python engineering
Azure
DevOps
Data modelling
Data quality
&nbsp;
Must have delivered at least one of:
<ul>
<li>A production Databricks pipeline with Unity Catalog row-level security in a regulated or finance setting.</li>
<li>A file-based, event-triggered ingestion framework with manifest or control-file validation.</li>
<li>A financial planning or EPM data integration into a lakehouse or warehouse.</li>
</ul>
Preferred skills, qualifications and attributes
Nice to have
<ul>
<li>Enterprise planning / EPM platforms: multi-dimensional cubes, dimensions and hierarchies, rules and consolidations, member-level security, and exporting data from them.</li>
<li>FP&amp;A domain knowledge: management P&amp;L, budget and forecast versions, adjustments and allocations, spend by product or project, GAAP vs non-GAAP.</li>
<li>Power BI with Databricks: DirectQuery, Entra SSO pass-through, impact of import models on row-level security.</li>
<li>Lakeflow Declarative Pipelines (DLT) and expectations, Auto Loader, file events on external locations.</li>
<li>Databricks system tables for cost and job monitoring; Lakeview dashboards and SQL alerts.</li>
<li>Experience in regulated environments (SOX controls, audit trails, change management).</li>
</ul>
Certifications (any is a plus)
<ul>
<li>Databricks Certified Data Engineer Professional</li>
<li>Databricks Certified Data Engineer Associate</li>
<li>Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Expert</li>
</ul>