Loop Through Large CSV?

Im trying to loop through a 145mb csv file but I keep getting Fatal error: Allowed memory size of xxxxxxxxxx bytes exhausted (tried to allocate 71 bytes)

I've tried setting the memory limit to 512m, 1024m, 2048m

while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
  if (in_array($data[0], array(
              1, 4, 6, 40, 50, 136, 451, 933, 1529, 2006, 2155, 2271, 2603, 2640,
              3173, 3619, 3789, 4479
          ))) {
    continue;
  }
  $in_data[] = $data;
}

Whats the best way to do this? What am I not understanding here? Any examples would help.