Section I) Websockets

Less than a week ago, we started discussing how to put -core on the web (our LOST countdown to start out with). There is a jungle out there, different protocols, mechanisms libraries and frameworks to dig through, the list is endless.

I failed miserably to find a working solution (just to know how to setup and use existing libraries was in the end much more difficult than to solve the problem at hand).

In the end, I ended up writing a Websocket code from scratch (largely based on the protocol description from wikipedia). The current implementation is < 200 lines of -core code.

A minimal application server in -core is given below:
// TestSocket.core
// Per Lindgren (C) 2014
//
// Example showing the use of websocket send/receive

include "Websocket.core"

Task client_receive(char_p msg) {
	#> printf("client wrote : %s\n", msg); <#
}

Task periodic() {
	#>
	static char send_buff[256];
	static int i = 0;
	sprintf(send_buff, "Counter %d", i++);
	<#
	sync ws_send(send_buff);
	async after 2s periodic();
}

Idle {
	#> printf("User Idle\n"); <# 
	async periodic();
	sync idle_websocket(); 
}


The corresponding web page embeds javascript to connect, disconnect, receive and display messages (from the application server) and take user input and send it to the application server.
<!DOCTYPE html>
<html lang="en">
<head>
  <title>WebSocket Echo Client</title>
  <meta charset="UTF-8" />
  <script>
    "use strict";
    // Initialize everything when the window finishes loading
    window.addEventListener("load", function(event) {
      var status = document.getElementById("status");
      var url = document.getElementById("url");
      var open = document.getElementById("open");
      var close = document.getElementById("close");
      var send = document.getElementById("send");
      var text = document.getElementById("text");
      var message = document.getElementById("message");
      var socket;

      status.textContent = "Not Connected";
      url.value = "ws://localhost:5000";
      message.textContent = "No data yet received";
      close.disabled = true;
      send.disabled = true;

      // Create a new connection when the Connect button is clicked
      open.addEventListener("click", function(event) {
        open.disabled = true;
        socket = new WebSocket(url.value, "lost-protocol");

        socket.addEventListener("open", function(event) {
          close.disabled = false;
          send.disabled = false;
          status.textContent = "Connected";
        });

        // Display messages received from the server
        socket.addEventListener("message", function(event) {
          message.textContent = "Server Says: " + event.data;
        });

        // Display any errors that occur
        socket.addEventListener("error", function(event) {
          //message.textContent = "Error: " + event;
          message.textContent = "Error: " + event.data;
        });

        socket.addEventListener("close", function(event) {
          open.disabled = false;
          status.textContent = "Not Connected";
        });
      });

      // Close the connection when the Disconnect button is clicked
      close.addEventListener("click", function(event) {
        close.disabled = true;
        send.disabled = true;
        message.textContent = "";
        socket.close();
      });

      // Send text to the server when the Send button is clicked
      send.addEventListener("click", function(event) {
        socket.send(text.value);
        text.value = "";
      });
    });
  </script>
</head>
<body>
  Status: <span id="status"></span><br />
  URL: <input id="url" /><br />
  <input id="open" type="button" value="Connect" />&nbsp;
  <input id="close" type="button" value="Disconnect" /><br />
  <input id="send" type="button" value="Send" />&nbsp;
  <input id="text" /><br />
  <span id="message"></span>
</body>
</html>


For the implementation of Websocket.core, I found the following link useful (after quite some digging).
https://developer.mozilla.org/en-US/docs/WebSockets/Writing_WebSocket_server

Notice
  • The use of openssl is there just for SHA1 and Base64Encode. In the future it would be great to have an implementation without external dependencies (other than those of the network stack).
  • The Base64Encode work on streams, the current implementation uses a memory mapped file to that end. Under OSX fmemopen is not native, and has to be emulated. I used the "hack" provided by Nimbuskit https://github.com/NimbusKit/memorymapping/.

To compile I use
gcc RTFM-PT.c -lpthread -lssl -lcrypto -I /usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib fmemopen.c -o PTCORE 

It assumes fmemopen.* to be in there RTFM-PT directory. For simplicity I also changes the path to autogen.c (in RTFM-PT.c) so that also autogen.c is the RTFM-PT directory, but the build structure can be improved on... If you compile through eclipse, be sure that the flags are set in you project:
  • -lpthread -lssl -lcrypto
  • -I /usr/local/opt/openssl/include
  • -L/usr/local/opt/openssl/lib
I used the "port" version of openssl, not the one distributed with OSX, hence the path /usr/local/opt/openssl/

To get debugging info, you may apply the option:
-D TRACE_WS


It has been tested only under OSX, for linux based systems it should work but is not tested (not sure exactly about linking for fmemopen and openssl but you will figure it out). Windows port will follow...

To try it locally, save the file (client.html) in your file system and open it through a browser. Compile and run the TestSocket.core application. Address for the web socket is default to your local machine at port 5000 but can be changed of course if you wish another port, or have a public ip for the computer your TestSocket.core is running. Websocket.core defines the address and port number, as seen below. INADDR_ANY binds the socket to all available interfaces, so if the computer has a public ip, it should be possible to connect (given that the port is made available to external connections).

This is NOT a course in computer communication, and we will help You get going for the LOST countdown challenge. Please let us know, if you run into any problems!

  serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
  serv_addr.sin_port = htons(5000);


The complete code is given below:

// Websocket.core
// Per Lindgren (C) 2014
//
#>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <string.h>
#include <openssl/sha.h>
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <math.h>
#include "fmemopen.h"

#ifdef TRACE_WS
#define DPS(fmt, ...) {fprintf(stderr, "\tWS:<%f> "fmt"\n", RT_time_to_float(time_get()), ##__VA_ARGS__);}
#else
#define DPS(...) 
#endif

typedef char* char_p;

// Encodes a string to base64
int Base64Encode(const char* message, int len, char** buffer) { 
  BIO *bio, *b64;
  FILE* stream;
  int encodedSize = 4 * ceil((double) len / 3);
  *buffer = (char *) malloc(encodedSize + 1);

  stream = fmemopen(*buffer, encodedSize + 1, "w");
  b64 = BIO_new(BIO_f_base64());
  bio = BIO_new_fp(stream, BIO_NOCLOSE);
  bio = BIO_push(b64, bio);
  BIO_set_flags(bio, BIO_FLAGS_BASE64_NO_NL); // write everything in one line
  BIO_write(bio, message, len);

  BIO_flush(bio);
  BIO_free_all(bio);
  fclose(stream);

  return (0); //success
}

int connfd = 0;
void error(char* err) {
  fprintf(stderr, "%s", err);
  exit(0);
}

char* resp1 = "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n";
char* resp2 = "Sec-WebSocket-Accept: ";
char* resp3 = "Sec-WebSocket-Protocol: lost-protocol\r\n\r\n"; // with an extra blank line
char* magic = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";

char sendBuff[1025];
<#
Func void idle_websocket() {
  #>
  int listenfd = 0;
  struct sockaddr_in serv_addr;

  char readBuff[1025], *reqline[3];

  listenfd = socket(AF_INET, SOCK_STREAM, 0);
  memset(&serv_addr, (char ) '\0', sizeof(serv_addr));
  memset(sendBuff, (char ) '\0', sizeof(sendBuff));
  memset(readBuff, (char ) '\0', sizeof(readBuff));
  serv_addr.sin_family = AF_INET;
  serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
  serv_addr.sin_port = htons(5000);

  bind(listenfd, (struct sockaddr*) &serv_addr, sizeof(serv_addr));
  listen(listenfd, 1);
  while (1) {
    DPS("before accept\n");
    connfd = accept(listenfd, (struct sockaddr*) NULL, NULL);
    DPS("after accept\n");
    int n = recv(connfd, readBuff, sizeof(readBuff), 0);
    if (n < 0)
      error("ERROR reading from socket");

    DPS("%d, %s\n", n, readBuff);reqline[0] = strtok(readBuff, " \t\n");
    if (strncmp(reqline[0], "GET\0", 4) == 0) {
      reqline[1] = strtok(NULL, " \t");
      reqline[2] = strtok(NULL, " \t\n");
      if (strncmp(reqline[2], "HTTP/1.1", 8) != 0) {
        write(connfd, "HTTP/1.0 400 Bad Request\n", 25);
        error("bad request");
      } 
      DPS("OK request\n");
      while (1) {
        reqline[0] = strtok(NULL, " \n\r");
        char ws_key[] = "Sec-WebSocket-Key:";
        DPS("%s\n", reqline[0]);if (strncmp(reqline[0], ws_key, sizeof(ws_key)) == 0) {
          reqline[1] = strtok(NULL, " \r\n");
          DPS("KEY = %s\n", reqline[1]);break;
        }
      }
      DPS("--- response header ---\n");
      send(connfd, resp1, strlen(resp1), 0);// the response header
      DPS("%s", resp1);

      // compute key
      char key_in[256];
      char* key_out;

      sprintf(key_in, "%s%s", reqline[1], magic); // append the magic
      DPS("key_in  %d : %s", (int) strlen(key_in), key_in);

      unsigned char hash[SHA_DIGEST_LENGTH];
      SHA1((const unsigned char *)key_in, strlen(key_in), hash); // SHA1 hashing
      DPS("hash, %d, %s", (int) sizeof(hash), hash);

      Base64Encode((const char *)hash, sizeof(hash), &key_out); // Encode as Base64
      DPS("key_out %d, %s", (int)strlen(key_out), key_out);

      sprintf(sendBuff, "%s%s\r\n", resp2, key_out);
      send(connfd, sendBuff, strlen(sendBuff), 0); // the unique session key
      DPS("%s", sendBuff);

      send(connfd, resp3, strlen(resp3), 0); // the protocol name
      DPS("%s", resp3);

      while (1) {
        DPS("--- recv ---");
        n = recv(connfd, readBuff, sizeof(readBuff), 0);
        if (n < 0)
          error("ERROR reading from socket");

        DPS("readBuff[0] %x ", (0xFF & readBuff[0]));
        unsigned char msg_fin = readBuff[0] & 0x80; // logic (bitwise) and

        if (msg_fin == 0)
          error ("message split, not implemented");

        DPS("fin OK");
        unsigned char msg_op = readBuff[0] & 0xF; // opcode 4 bits

        if (msg_op == 0x8) {
          DPS("disconnect by server");
          break;
        }

        if (msg_op != 0x1)
          error ("non-text message, not implemented");
        DPS("Text msg OK");

        unsigned char msg_size = readBuff[1] & 0x7F; // length
        if (msg_size >= 126)
          error ("multi byte length, no implemented");
        DPS("Size OK %d", msg_size);

        unsigned char *decoded = (unsigned char *) &readBuff[6];
        unsigned char *encoded = (unsigned char *) &readBuff[6];
        unsigned char *mask = (unsigned char *) &readBuff[2]; // point to the mask bits

        for (int i = 0; i < msg_size; i++) 
          decoded[i] = (encoded[i] ^ mask[i % 4]);

        decoded[msg_size] = 0; // terminate the string
        DPS("Text msg %s", decoded);

        <# async client_receive((char_p)decoded); #>
      }
      DPS("trying to reconnect");
      close(connfd);
    }
  }
  // never happens	
  close(listenfd);
  <#
}

Func void ws_send(char_p message) {
  #>
  if (connfd == 0)
    return;

  unsigned char *out_decoded = (unsigned char *) &sendBuff[2]; //6 with mask
  sprintf((char *) out_decoded, "%s", message);

  int len = strlen(message);
  if (len > 126)
    error("we do not support split messages\n");
  sendBuff[0] = 0x80 | 0x1; // FIN + text_msg opcode
  sendBuff[1] = 0x00 | len; // no mask
  send(connfd, sendBuff, len + 2, 0);
  <#
}

Per Lindgren, founder of RTFM-lang, 2014

Last edited Sep 10, 2014 at 8:30 PM by RTFMPerLindgren, version 12