elearn: Streaming CSV Export for a 180,000-User Student Portal
Scaled a Django student-portal export path from an in-memory bottleneck to an async, streaming pipeline handling 2.5M rows, and shipped permission-gated, audited impersonation for support workflows.
Context
elearn is Code Argo’s Django student portal for the BITS Pilani ecosystem, and the largest, longest-running codebase I work on: roughly 180,000 total users, about 75,000 of them active students, under continuous development for over a year. Another engineer laid the foundation and remains a major contributor; I owned frontend development end to end early on, and now lead new features, improvements, and fixes.
Exporting MGPA and student-evaluation datasets to CSV is a routine reporting need. As enrollment and the course catalog grew, that export path stopped keeping up.
Constraints
- The export has to cover an entire cohort or semester of evaluation data in one pass; its size scales with enrollment, not with a fixed page limit.
- Any change had to fit the existing Django request/response cycle and the staging-branch release workflow without destabilizing features shipping in parallel.
- It had to stay genuinely usable for staff running real reports, not merely avoid crashing. An export that timed out under a browser request was no better than one that failed.
Architecture decision
The original path built the entire response in memory before sending a byte. I replaced it with an async, streaming implementation that pulls and writes rows incrementally, so response time and memory footprint no longer scale with dataset size. That raised the safe export ceiling to 2.5 million rows with no change to how staff use the feature.
Alongside it, I shipped user impersonation (“Log in as”) for support staff, so support can reproduce student-reported issues in the student’s own context without losing traceability. Every session is audit-logged. Access started superuser-only and later widened to anyone holding the right permission, with explicit guards against using it to escalate privilege, and the whole path is covered by Django tests.
Outcome
The streaming export now serves datasets up to 2.5 million rows that the in-memory version couldn’t reliably handle, and it landed without disrupting the parallel feature stream. Impersonation gives support an audited, permission-gated way to debug on a student’s behalf. Both came out of the feature ownership I took on after moving off frontend-only work.