Data Cleaning & Processing
Dirty data quietly ruins good decisions. I clean, validate and structure your datasets so the analysis, models and reports built on top of them can be trusted.
Duplicate customers inflate your revenue. Three date formats break every time-series. A revenue column stored as text breaks every calculation downstream. None of it throws an error — it just produces a confident, wrong answer.
What you get
- Deduplication with rules you agreed toA matching key defined with you, then survivorship rules for which value wins when two records disagree. Nothing is merged on a guess.
- One format per fieldDates to ISO 8601, currency to a defined standard, phone numbers and addresses to a consistent shape, categories mapped to a controlled list.
- Missing values handled deliberatelyDropping rows is one option among several, and rarely the right one. Every imputation choice is recorded with its reasoning.
- Nothing deleted, everSuspect rows are quarantined rather than removed, so a judgement call can be reversed after somebody who knows the business looks at it.
- Repeatable, not a one-off rescueDelivered as a script or scheduled pipeline, so next quarter's data arrives clean instead of needing the same rescue again.
My Data Cleaning & Processing process
Profile the dataset
Types, distributions, null rates, duplicate counts, outliers. This produces a written data-quality report before anything is changed.
Agree the rules
Which records are the same record, which value wins, what is out of range. These are business decisions, so they are yours to make.
Clean reproducibly
Python and version control. Every run on the same input produces the same output, and the transformation log says what changed and why.
Validate the result
Automated checks plus a sampled manual review, with before-and-after counts on every rule applied.
Automate the recurrence
Schedule it, add quality thresholds, and alert when incoming data breaches them.
Tools I use for this
Chosen per project against your constraints and budget — never one stack forced onto every problem.
Questions I get asked about this
What formats can you take, and how do I send the data?
CSV, Excel, Google Sheets, JSON, XML, Parquet, SQL dumps, and direct read-only database connections. For files, a shared drive folder is fine. For anything containing personal data I would rather take a read-only database user or an encrypted transfer than an email attachment, and I will set that up with you. Fixed-width and PDF-trapped tables are also possible; they take longer, so it is worth flagging early if that is what you have.
Will you delete rows I might turn out to need?
No. Nothing is deleted. Rows that fail a rule are moved to a quarantine table with the reason attached, so somebody who knows the business can look at them and decide. This matters more than it sounds: outliers are often either a typo with an extra zero or your single most valuable customer, and only somebody with context can tell those apart. You also get a full transformation log, so any rule can be reversed and re-run.
Is this a one-time clean or something ongoing?
Either, and the choice usually depends on where the mess comes from. If it is historic — a migration, a merger, years of accumulated free-text entry — a one-time clean genuinely finishes the job. If new records arrive dirty every week, cleaning the backlog only buys a few months, and the durable fix is validation at the point of entry plus a scheduled pipeline. I will tell you which of the two you have after profiling.
How do you decide which record wins when two duplicates disagree?
With survivorship rules that you approve before anything runs. Common ones are most-recently-updated wins, most-complete record wins, or a trusted source beats a less trusted one — and often it is field-by-field, taking the newest phone number but the oldest signup date. Where records conflict in a way no rule can settle, the pair is flagged for review rather than silently resolved. Bad merges are far harder to detect later than unmerged duplicates.
Clean data, clear decisions
Send a sample file — a few hundred rows is plenty. I will send back a profile of what is actually wrong with it before you commit to anything.
Start with a free discovery call