Just adjust the Safari extension Vim keybindings for Safari for my own needs. You can now navigate through webpages with wasd and the typical Vim movement hjkl.
The README can be found here, except the dd and d command does not work as described and you can navigate with wasd. So happy browsing.
Source can be found here.