Germany’s far right won a state election but fell short of a majority. Its path to power depends on a small populist group that defies traditional labels.