Pandas 迁移 Polars 的代码方案
盘点 pandas 用法并逐步改写为 Polars 表达式,处理空值与顺序差异,以对等测试和基准测试验证迁移。
完整提示词
<context>
Polars is not a drop-in pandas replacement, and a line-by-line port usually ends slower than the original. Polars has no row index: anything that relies on `set_index`, `loc`, index alignment in arithmetic, `reset_index` or MultiIndex results must be rewritten with explicit key columns and joins. Speed comes from expressions (`pl.col(...)`, `with_columns`, `group_by().agg(...)`, `when/then/otherwise`) and from the lazy API (`scan_parquet`, `scan_csv`, `.lazy()` then `.collect()`), which lets the query optimizer push down filters and projections. `map_elements` and Python UDFs bring back row-by-row speed and should be a last resort. Semantics differ in ways that change results silently: Polars keeps null distinct from NaN, aggregations skip nulls, joins do not sort, `group_by` does not keep group order unless `maintain_order=True`, string and datetime dtypes are strict and casting failures raise, and integer columns with nulls stay integers instead of becoming float. Interop is cheap through Arrow (`pl.from_pandas`, `.to_pandas()`), so a boundary at the edges is fine while the core moves.
</context>
<task>
Scope: {{scope}}
{{#constraints}}Must stay on pandas or accept pandas at the boundary: {{constraints}}{{/constraints}}
1. Read the code and inventory pandas usage: index and MultiIndex reliance, `apply`/`iterrows`/`transform`, groupby patterns, merges, reshapes (pivot, melt, stack), datetime and string accessors, IO calls, and every place a DataFrame crosses into another library.
2. Before changing anything, capture reference outputs: run the current code on representative inputs (including nulls, empty frames and duplicate keys) and save the results for comparison. If no tests exist, write parity tests first.
3. Port one function or step at a time to expression style. Replace index logic with explicit key columns, `apply` with expressions, and eager reads with `scan_*` plus a single `.collect()` where the pipeline allows. Keep conversions to pandas only at boundaries named in step 1.
4. Handle semantic differences on purpose: decide null versus NaN handling, add `maintain_order=True` or an explicit sort where order matters, set dtypes explicitly on read, and note any changed output column order or names.
5. Run the parity tests after each step, comparing values, dtypes, row order (where it matters) and null counts. Then time the old and new versions on realistic data and report memory where relevant.
</task>
<constraints>
- Never port `apply` or `iterrows` into `map_elements` when an expression exists; if no expression fits, say so and explain the cost.
- Do not claim a speed-up without a measured before-and-after on the same input.
- Keep the old pandas path runnable until parity is shown, and do not change business logic in the same diff.
{{> guardrails/verify-before-done}}
{{> guardrails/scope-discipline}}
</constraints>
<output_format>
## Inventory
Table: location | pandas pattern | Polars approach | risk.
## Translation notes
Semantic differences that affected this code and how each was handled.
## Changes made
Per step: what changed, lazy or eager, boundary conversions kept.
## Parity results
Test results and before-and-after timings with data size.
## Left on pandas
What stays and why, or "Nothing".
</output_format>可替换内容
- {{scope}}
- {{#constraints}}
- {{constraints}}
- {{/constraints}}
- {{> guardrails/verify-before-done}}
- {{> guardrails/scope-discipline}}
分类与标签
编程开发、Pandas 迁移、Polars、数据处理、性能基准