Crawling behind Login-Form

Questions/Help
, ,
#1

Hello Flutter Devs!

I am working on small app that logs into my university account and downloads my timetable schedule.
university.com/login site is made up of two field login form. There is no CSRF token. Session Cookie is sent after login.

I tried sending post request to this site with provided credentials using http library client without succeeding.
After digging in Chrome dev tools I discovered that the post request made by Chrome includes three cookies that are put on my browser when I visit the site. Nevertheless, these cookies were invisible for my poor crawler implemented using http library (didnt get these back when visited site with crawler).
I ended up importing puppeteer library and I tried using its headless browser to capture these hidden cookies and putting them on post request sent again by http client. Didnt work either.
In the end I implemented whole crawler using the puppeteer library which works but I think it is an overkill for this simple case.

Below is the crawler version that got me to the point where I was able to receive 200 OK response. I am capturing pre-login/hidden cookies using puppeteer headless browser and feeding these into post request with credentials and other headers send by http client. Unfortunately, there was no Session Cookie, nor I got redirected further.

import 'package:http/http.dart';
import 'package:puppeteer/puppeteer.dart' as headlessBrowser;
import 'package:puppeteer/protocol/network.dart' as headlessCookie;

void main() async {

  var headers = {
    "Host":                      "university.com",
    "Connection":                "keep-alive",
    "Content-Length":            "171",
    "Cache-Control":             "max-age=0",
    "Origin":                    "https://university.com",
    "Upgrade-Insecure-Requests": "1",
    "Content-Type":              "application/x-www-form-urlencoded",
    "User-Agent":                "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/79.0.3945.79 Chrome/79.0.3945.79 Safari/537.36",
    "Sec-Fetch-User":            "?1",
    "Accept":                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "Sec-Fetch-Site":            "same-origin",
    "Sec-Fetch-Mode":            "navigate",
    "Referer":                   "https://university.com/system/login.pl?logout=1;lang=en-gb",
    "Accept-Encoding":           "gzip, deflate, br",
    "Accept-Language":           "en-gb;q=0.8, en;q=0.7"
  };
  var body = {
    "lang":           "en-gb",
    "login_hidden":   "1",
    "destination":    "/auth/?lang=en-gb",
    "auth_id_hidden": "0",
    "auth_2fa_type":  "no",
    "credential_0":   "JohnDoe",
    "credential_1":   "JohnDoesPassword",
    "credential_k":   "",
    "credential_2":   "86400"
  };

  var browser = await headlessBrowser.puppeteer.launch();
  var myPage = await browser.newPage();
  await myPage.goto("https://university.com", wait: headlessBrowser.Until.networkIdle);
  List<dynamic> listOfDynamic = await myPage.cookies(urls: ["https://university.com"]);
  List<headlessCookie.Cookie> cookies = listOfDynamic.cast<headlessCookie.Cookie>().toList();
  String cookiesSendFormat = formatCookies(cookies);
  headers.putIfAbsent("Cookies", () => cookiesSendFormat);

  var client = new Client();
  Response response = await client.post("https://university.com/system/login.pl",
          headers: headers,
          body: body);
  client.close();
  await browser.close();
}

Questions:

  1. Is there a way to login using plain http library?
  2. Did I miss something that is usually required by website to login?
  3. How bad of a solution is using headless browser?
  4. What would you do in my place?

Thank you!