A dynamic influence in artistic circles, the Russian-born painter devised a haunting idiosyncratic style of her own